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Objectives 

The  objective  of  the  project  was  to  develop  ACT-R  models  of  student  use  of  computer-based 
instruction.  The  principal  focus  of  our  research  was  developing  a  model  of  student  interaction 
with  Valerie  Shute's  Stat  Lady.  We  wanted  to  learn  information  about  the  appropriateness  of 
ACT-R  for  this  kind  of  project,  evaluate  different  methodologies,  and  make  suggested 
improvements  to  Stat  Lady.  This  effort  required  collection  of  detailed  subject  verbal  protocols, 
development  of  an  ACT-R  simulation  of  student  behavior,  and  validation  of  that  simulation 
against  the  large  corpus  of  data  collected  at  the  Armstrong  Human  Resources  Laboratory.  We 
then  used  the  cognitive  model  to  suggest  variations  on  the  Stat  Lady  system  and  assessed  these  in 
experiments  at  Train  Laboratory  at  Lackland  Air  Force  Base.  We  also  investigated  how 
informative  eye  movement  protocols  were  relative  to  verbal  protocols. 

Research  Summary 

We  videotaped  and  transcribed  the  interactions  of  four  students  working  with  the  descriptive 
Statistics  module  of  Stat  Lady.  For  two  of  the  sections  in  this  module  (Simple  Frequency 
Distributions  and  Grouped  Frequency  Distributions)  we  coded  all  the  interactions  and  identified 
all  the  learning  events  (where  particular  curriculum  objectives  were  learned).  We  validated  this 
detailed  data  against  computer  records  of  180  students  from  the  TRAIN  laboratory  at  Armstrong. 
We  focused  on  modeling  student  interactions  in  the  scenarios  within  a  section  because  this  is 
where  students  spend  most  of  their  time  and  this  is  where  most  of  the  effort  has  gone  into  the 
design  of  Stat  Lady.  We  showed  that  subjects  speed  up  both  because  they  read  less  of  the 
instruction  and  because  they  speed  up  in  the  individual  steps  of  scenario  interaction.  We 
developed  an  ACT-R  model  that  read  the  instruction  and  performed  the  individual  steps  of  the 
scenarios.  It  learned  to  skip  those  parts  of  the  instruction  that  are  uninformative  and  memorized 
other  parts  of  the  instruction.  For  both  reasons  it  was  able  to  model  subjects’  drop  out  of 
reading.  It  also  strengthened  its  knowledge  representation  and  for  this  reason  came  to  perform 
the  individual  steps  more  rapidly.  It  was  the  first  detailed  cognitive  model  of  the  process  of 
interacting  with  instructional  software.  We  have  subsequently  validated  this  model  against  a 
larger  population  of  subjects  at  the  TRAIN  laboratory.  We  also  used  it  to  make  a  set  of 
suggested  modifications  to  Stat  Lady  which  have  also  been  validated  in  a  new  at  the  TRAIN  lab. 

We  developed  an  ACT-R  model  of  an  student  which  can  interact  with  a  LISP-based  emulation  of 
Stat  Lady.  The  simulation  was  able  to  reproduce  the  learning  of  the  Stat  Lady  subjects.  It 
identified  much  of  this  learning  as  interface  driven.  It  also  suggested  that  much  of  the  interaction 
in  Stat  Lady  does  not  contribute  to  learning  of  curriculum  objectives.  As  such,  it  suggested  that 
we  would  produce  a  much  “leaner”  version  of  Stat  Lady  which  would  accomplish  the  same 
curriculum  objectives.  We  developed  an  alternate  version  of  Stat  Lady  that  did  not  involve  as 
elaborate  feedback  or  as  rich  scenarios.  In  an  assessment  at  the  TRAIN  Lab  we  showed  that  this 
version  of  the  tutor  was  as  successful  in  achieving  curriculum  objectives,  did  so  in  less  time,  and 
without  sacrificing  student  measures  of  enjoyment.  This  is  a  demonstration  of  how  a  cognitive 
micro-analysis  of  a  system  can  lead  to  the  design  of  a  more  effective  system. 
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The  final  project  was  to  assess  the  suitability  of  eye  movements  to  provide  information  about  the 
nature  of  the  interaction  between  student  and  tutor.  In  this  project  we  used  the  PAT  algebra  tutor 
that  has  been  used  Carnegie  Mellon  University.  Here  we  showed  that  student  eye  movements 
could  serve  to  predict  when  they  were  going  to  make  errors  and  to  disambiguate  the 
interpretation  of  various  actions.  We  also  found  that  they  informed  us  about  when  students 
actually  processed  help  messages  and  error  messages  presented  by  the  tutor.  In  his  dissertation, 
Gluck  identified  a  number  of  instructional  opportunities  afforded  by  eye  movements.  He  also 
developed  an  ACT-R  model  of  interaction  with  the  tutor  that  predicted  eye  movements. 

Personnel  Support  over  the  History  of  the  Grant 

1 .  John  Anderson,  Principal  Investigator 

2.  Kevin  Gluck,  Graduate  Student 

3.  Marsha  Lovett,  Research  Scientist 

4.  Daniel  Bothell,  Research  Programmer 

5.  Joon  Park,  Undergraduate  Research  Assistant 

6.  Serene  Taleb-Agha,  Research  Programmer 
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Interactions/Transitions 


We  have  established  contact  with  Wes  Regian  and  particularly  Valerie  Shute  at  Brooks  Air  Force 
Base.  Valerie  Shute  visited  CMU  in  May  of  1 996  and  developed  plans  for  collaboration.  Kevin 
Gluck  visited  the  San  Antonio  labs  numerous  times.  He  and  Valerie  Shute  collaborated  on  the 
redesign  of  Stat  Lady  and  its  testing  in  the  TRAIN  laboratory.  He  made  a  presentation  of  the 
modeling  work  on  ACT-R  at  the  Fourth  Annual  ACT-R  Workshop  and  at  the  ITS  98.  Marsha 
Lovett  gave  a  presentation  at  the  American  Statistical  Association  on  evaluation  of  Statistics 
instruction. 


Honors/ Awards 

1994  American  Psychological  Association’s  Distinguished  Scientific  Career  Award 

1 999  National  Academy  of  Sciences 
1 999  Fellow,  American  Academy  of  Arts  and  Sciences 
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